When an image is gotten (i.e., captured) with a digital camera, noise may be added to the image depending on a characteristic of its own CCD (charge-coupled device) or CMOS read circuit or a property of the transmission line. In addition, the image may also get blurred due to either out-of-focus shooting or a camera shake. In this manner, the quality of an image shot may deteriorate due to not only the noise resulting from the image's own characteristic but also the blur that has been caused by hand during the shooting operation. Among these multiple different kinds of “blurs”, the blur of an image that has been induced by the movement of a camera during shooting (or exposure) will be sometimes referred to herein as a “motion blur”.
Recently, as there is a growing demand for high-sensitivity shooting, it has become more and more necessary to restore such a blurry deteriorated image (which will be simply referred to herein as a “deteriorated image”) to the original condition (which will be referred to herein as an “ideal image”) as perfectly as possible. To capture a bright and noise- or blur-free image that should be obtained by high-sensitivity shooting, there are roughly two approaches: one of them is to increase the sensitivity and the other is to extend the exposure time.
If the sensitivity is increased, however, the noise will be amplified, too. As a result, the signal will be drowned in the noise, and the resultant image will be mostly covered with the noise in many cases. On the other hand, if the exposure time is extended, the light produced on the spot can be accumulated a lot and the resultant image will have much less noise. In that case, the signal will not be drowned in the noise but the image will get shaky much more easily due to a camera shake, which is a problem.
Thus, in order to extend the exposure time, two different approaches have been taken so far. One of them is optical image stabilization such as a lens shift or a sensor shift. According to the other method, the direction and/or the magnitude of the motion blur is/are obtained based on the captured image or by the sensor and then subjected to signal processing, thereby restoring the image (which will be referred to herein as a “restoration method by signal processing”).
The optical image stabilization has only a limited correction range. If the exposure time is extended, a camera shake will be produced more easily, and therefore, the lens or the sensor needs to be shifted in a broader range. However, the longer the distance the lens or sensor needs to go, the longer the time delay caused by their shift. In addition, a physical limit prevents the designer from increasing the size of the camera unlimitedly.
Meanwhile, the restoration methods by signal processing are disclosed in Patent Document No. 1 and Non-Patent Documents Nos. 1 and 2, for example. Hereinafter, those restoration methods by signal processing will be described.
Suppose the intensity distribution of an image produced on the imaging area of an image sensor is represented by I(x, y). The coordinates (x, y) are two-dimensional coordinates indicating the location of a pixel (i.e., a photosensitive cell) on the imaging area. For example, if the image is made up of M×N pixels that are arranged in columns and rows and if x and y are integers that satisfy the inequalities 0≦x≦M−1 and 0≦y≦N−1, respectively, then the location of each of the pixels that form the image can be indicated by a set of coordinates (x, y). In this example, the origin (0, 0) of the coordinate system is supposed to be put at the upper left corner of the image, the x-axis is supposed to run vertically and the y-axis is supposed to run horizontally. However, the coordinate system may also be defined in any other way.
If the intensity distribution of a blur-free image (which is an ideal image or what the image should be) is represented by L(x, y), a point spread function (PSF) that defines the blur is represented by PSF(x, y) and the noise is represented by n(x, y), then the following Equation (1) is satisfied:I(x,y)=PSF(x,y)*L(x,y)+n(x,y)  (1)where the sign * indicates a two-dimensional convolution operation.
The point spread function PSF(x, y) of a camera shake depends on the trace of the camera that has been shaken (which will be referred to herein as a “camera shake trace”) during shooting (or exposure). Since the camera shake trace changes every time an image is shot with the camera, PSF(x, y) also changes every shooting session with the camera.
If the camera shake trace during shooting has been detected by a gyrosensor, for example, and if PSF(x, y) is known, the deteriorated image I(x, y) can be restored to an image L(x, y) by performing a deconvolution operation using that PSF(x, y). On the other hand, unless PSF(x, y) is known, the deteriorated image I(x, y) needs to be restored to the image L(x, y) by estimating PSF(x, y) based on the deteriorated image I(x, y). The former method is called a “non-blind deconvolution”, while the latter method is called a “blind deconvolution”. Since PSF(x, y) and the image L(x, y) both need to be estimated based on the deteriorated image I(x, y) according to the blind deconvolution method, it is more difficult to reduce the blur than when the non-blind deconvolution method is adopted.
A PSF convolution operation that defines a camera-shake-induced motion blur is performed with a linear filter. A two-dimensional convolution operation linear filter is usually represented by a kernel consisting of a coefficient matrix with a size of N×N pixels, where N is an integer that is equal to or greater than three. The PSF that defines a blur can be represented by a blur kernel. Thus, to restore a blurry image to a less blurred (or non-blurred) image, the blur kernel that defines the blur needs to be estimated.
As a method for restoring an image by signal processing, Non-Patent Document No. 1 discloses that a multi-scale inference scheme is used to estimate a blur kernel and a restored image based on a single blurry image. According to that multi-scale inference scheme, first of all, a blur kernel having a size of 3×3 pixels is estimated by using a deteriorated image with a low resolution. After that, by gradually increasing the resolution of the deteriorated image for use in estimation, the resolution of the blur kernel is also raised. FIGS. 18(a) through 18(h) illustrate the method disclosed in Non-Patent Document No. 1. The upper part of FIG. 18(a) shows a blur kernel of 3×3 pixels, while the lower part shows a restored image with a corresponding resolution. The same statement applies to FIGS. 18(b) through 18(h), in which the resolution increases gradually in this order.
As there are a small number of pixels in the deteriorated image with a low resolution, the size of pixels with a camera-shake-induced motion blur is also small. As a result, in the deteriorated image with a low resolution, the size of the blur kernel becomes small and the complexity of computations to do to estimate the blur kernel is also little. Moreover, if the blur kernel were estimated using a deteriorated image with a high resolution and a large number of pixels from the beginning, then the estimation could converge to a local minimal representing a different blur kernel from the true one. According to the multi-scale inference scheme, however, the blur kernel can be estimated more accurately.